fix(scheduler): load dag_run eagering for execution API validation by Lee-W · Pull Request #56916 · apache/airflow

Lee-W · 2025-10-21T01:13:11Z

Why

how to reproduce (through breeze on `e24cfb0`)

Add the following lines to files/airflow-breeze-config/environment_variables.env

AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT=1
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT_DETECTION_INTERVAL=5

Add the following dag to files/dags

Dag

from __future__ import annotations

import os
import time

import pendulum

from airflow.sdk import Asset, Metadata, dag, task

asset = Asset("test")


@dag(schedule=None, start_date=pendulum.datetime(2025, 1, 1), catchup=False)
def producer():
    @task(outlets=[asset])
    def producer_task():
        yield Metadata(asset=asset, extra={"test": 1})

    producer_task()


@dag(schedule=asset, start_date=pendulum.datetime(2025, 1, 1), catchup=False)
def consumer():
    @task()
    def consumer_task(ds_str):
        print(ds_str)
        time.sleep(5)
        os._exit(1)  # immediate exit, no cleanup, no heartbeat

    consumer_task("{{ (triggering_asset_events.values() | first | last).extra }}")


producer()
consumer()

breeze start-airflow --db-reset --backend postgres
Turn on both produer and conumser Dags
Continously trigger producer (10 ~ 20 times) really quick

What

Kaxil implemented a _eager_load_dag_run_for_validation method to ensure that dag_run exists even when a ti becomes a zombie task

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

kaxil

Were you able to reproduce the error & does this fix it. (It should but worth verifying)

Lee-W · 2025-10-21T01:26:44Z

Were you able to reproduce the error & does this fix it. (It should but worth verifying)

yep, I'm now simplifying the sample code I used and will add to the desciprion

Lee-W · 2025-10-21T01:41:57Z

Updated the Dag and steps I used in detail

…alidation (apache#56916) (cherry picked from commit 8ffb7b1) Co-authored-by: Wei Lee <weilee.rx@gmail.com>

…ation (apache#56916)" This reverts commit 8ffb7b1.

potiuk · 2025-10-21T10:00:17Z

Hey @Lee-W -> we had to revert it as it fails for Python 3.13. Can you please re-do it (and set all versions label on PR to trigger the tests for 3.13.

See https://github.com/apache/airflow/actions/runs/18676364216/job/53249204014

…ation (#56916)" (#56943) This reverts commit 8ffb7b1.

…ation (apache#56916)" (apache#56943) This reverts commit 8ffb7b1.

…on API validation (apache#56916)" (apache#56943) This reverts commit 8ffb7b1. (cherry picked from commit 209fb2e) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…on API validation (#56916)" (#56943) (#56956) This reverts commit 8ffb7b1. (cherry picked from commit 209fb2e)

…dation (apache#56916)" (apache#56943) This reverts commit 209fb2e.

…57010) * Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943) This reverts commit 209fb2e. * fix(scheduler): joinedload TI.dag_run instead of selectinload

…alidation (#57010) * Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943) This reverts commit 209fb2e. * fix(scheduler): joinedload TI.dag_run instead of selectinload (cherry picked from commit 833717a) Co-authored-by: Wei Lee <weilee.rx@gmail.com>

…alidation (#57010) (#57045) Co-authored-by: Wei Lee <weilee.rx@gmail.com> fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)

…alidation (apache#57010) (apache#57045) Co-authored-by: Wei Lee <weilee.rx@gmail.com> fix(scheduler): load dag_run eagering for execution API validation (apache#56916)" (apache#56943)

…57062) * [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031) Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com> * [v3-1-test] Fix Extra Links translation key. (#57030) (#57040) (cherry picked from commit 8d71c07) Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> * [v3-1-test] Fix dags list not rendering (#57037) (#57046) (cherry picked from commit 8668fe5) Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com> * [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045) Co-authored-by: Wei Lee <weilee.rx@gmail.com> fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943) * [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047) Fix dag list cache invalidation on favouriting a dag (cherry picked from commit 4e46212) Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com> * [v3-1-test] Add new PL translations. (#56359) (#57055) (cherry picked from commit 6583b6d) * Add last missing translation for 3.1.1 (#57057) * [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048) * [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) * fix: adjust PanelButtons spacing and alignment * Fix PanelButtons layout and button styling * Fix PanelButtons: use bg.subtle and remove unnecessary _hover (cherry picked from commit 4c35cd2) Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com> Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com> Co-authored-by: Wei Lee <weilee.rx@gmail.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>

… en (#57060) (#57064) * [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031) Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com> * [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#57062) * [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031) Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com> * [v3-1-test] Fix Extra Links translation key. (#57030) (#57040) (cherry picked from commit 8d71c07) Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> * [v3-1-test] Fix dags list not rendering (#57037) (#57046) (cherry picked from commit 8668fe5) Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com> * [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045) Co-authored-by: Wei Lee <weilee.rx@gmail.com> fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943) * [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047) Fix dag list cache invalidation on favouriting a dag (cherry picked from commit 4e46212) Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com> * [v3-1-test] Add new PL translations. (#56359) (#57055) (cherry picked from commit 6583b6d) * Add last missing translation for 3.1.1 (#57057) * [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048) * [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) * fix: adjust PanelButtons spacing and alignment * Fix PanelButtons layout and button styling * Fix PanelButtons: use bg.subtle and remove unnecessary _hover (cherry picked from commit 4c35cd2) Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com> Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com> Co-authored-by: Wei Lee <weilee.rx@gmail.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com> * [v3-1-test] Add possibility of removing extra translations missing in en (#57060) Sometimes we rename/move translation - we did not have a tool to remove such dangling translations. Now we have. (cherry picked from commit 2314099) Co-authored-by: Jarek Potiuk <jarek@potiuk.com> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com> Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com> Co-authored-by: Wei Lee <weilee.rx@gmail.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>

Lee-W requested review from XD-DENG and ashb as code owners October 21, 2025 01:13

boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label Oct 21, 2025

Lee-W requested review from kaxil and removed request for XD-DENG and ashb October 21, 2025 01:13

fix(scheduler): load dag_run eagering for execution API validation

d896184

Lee-W force-pushed the fix-asset-DetachedInstanceError branch from 6f35b7f to d896184 Compare October 21, 2025 01:17

kaxil approved these changes Oct 21, 2025

View reviewed changes

kaxil added this to the Airflow 3.1.1 milestone Oct 21, 2025

Lee-W merged commit 8ffb7b1 into apache:main Oct 21, 2025
60 checks passed

Lee-W deleted the fix-asset-DetachedInstanceError branch October 21, 2025 02:07

Lee-W added the backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch label Oct 21, 2025

Lee-W mentioned this pull request Oct 21, 2025

fix(scheduler): load dag_run eagering for execution API validation #56919

Merged

potiuk added a commit to potiuk/airflow that referenced this pull request Oct 21, 2025

Revert "fix(scheduler): load dag_run eagering for execution API valid…

4afd95c

…ation (apache#56916)" This reverts commit 8ffb7b1.

potiuk mentioned this pull request Oct 21, 2025

Revert "fix(scheduler): load dag_run eagering for execution API valid… #56943

Merged

potiuk added a commit that referenced this pull request Oct 21, 2025

Revert "fix(scheduler): load dag_run eagering for execution API valid…

209fb2e

…ation (#56916)" (#56943) This reverts commit 8ffb7b1.

Jonpaco23 pushed a commit to Jonpaco23/airflow that referenced this pull request Oct 21, 2025

Revert "fix(scheduler): load dag_run eagering for execution API valid…

9e491da

…ation (apache#56916)" (apache#56943) This reverts commit 8ffb7b1.

potiuk mentioned this pull request Oct 21, 2025

[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi… #56956

Merged

potiuk added a commit that referenced this pull request Oct 21, 2025

[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi…

5d8860f

…on API validation (#56916)" (#56943) (#56956) This reverts commit 8ffb7b1. (cherry picked from commit 209fb2e)

Lee-W added a commit to astronomer/airflow that referenced this pull request Oct 22, 2025

Reapply "fix(scheduler): load dag_run eagering for execution API vali…

dcb726b

…dation (apache#56916)" (apache#56943) This reverts commit 209fb2e.

Lee-W mentioned this pull request Oct 22, 2025

fix(scheduler): load dag_run eagering for execution API validation #57010

Merged

github-actions bot mentioned this pull request Oct 22, 2025

[v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) #57045

Merged

potiuk mentioned this pull request Nov 8, 2025

Add missing translations for PL for 3.1.3 #58076

Merged

Lee-W mentioned this pull request Dec 23, 2025

Gracefully Handle Invalid consumed_asset_event in DagRunContext #59740

Closed

1 task

SameerMesiah97 mentioned this pull request Dec 27, 2025

Added defensive "before" validator in DagRunContext to handle DetachedInstanceError #59857

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(scheduler): load dag_run eagering for execution API validation#56916

fix(scheduler): load dag_run eagering for execution API validation#56916
Lee-W merged 1 commit intoapache:mainfrom
astronomer:fix-asset-DetachedInstanceError

Lee-W commented Oct 21, 2025 •

edited

Loading

Uh oh!

kaxil left a comment

Uh oh!

Lee-W commented Oct 21, 2025 •

edited

Loading

Uh oh!

Lee-W commented Oct 21, 2025

Uh oh!

Uh oh!

potiuk commented Oct 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Lee-W commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

how to reproduce (through breeze on e24cfb0)

What

Uh oh!

kaxil left a comment

Choose a reason for hiding this comment

Uh oh!

Lee-W commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Lee-W commented Oct 21, 2025

Uh oh!

Uh oh!

potiuk commented Oct 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Lee-W commented Oct 21, 2025 •

edited

Loading

how to reproduce (through breeze on `e24cfb0`)

Lee-W commented Oct 21, 2025 •

edited

Loading